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An Evaluation of the New York School Choice 
Scholarships Program: The First Year 

(Executive Summary) 

The School Choice Scholarships Foundation (SCSF) announced in February 1997 
that it would provide 1,300 scholarships so that children from low-income families 
currently attending public schools could transfer to private schools. The scholarships 
were worth up to $1,400 annually and could be redeemed for at least three years at both 
religious and secular schools. SCSF received initial application forms from over 20,000 
students between February and late April 1997. Scholarship recipients were selected in a 
lottery held in May 1997 and began school the following fall. 

In this paper we report the first-year results for an evaluation of the SCSF 
program. The evaluation takes advantage of the fact that the SCSF lottery allowed for the 
conduct of a natural randomized experiment, in which students were allocated randomly 
to scholarship and control groups. 

Our major findings from a survey of scholarship parents and students, as 
compared to a similar group of families in the control group, are as follows: 

Outcomes: 

• After one year, students who received a scholarship scored higher in math and 
reading tests. Overall, the differences between all those in grades two through 
five who used their scholarships to attend a private school and the control 
group of students were small — around two percentile points in both subjects. 
However, sizeable differences were observed among those students who were 
in the fourth and fifth grades — four percentile points in reading and six points 
in math. 

• Parents of scholarship users are much more satisfied with their children’s 
education. Nearly half the scholarship users give their school an “A”, as 
compared to only one-eighth of the control group. Scholarship families were 
substantially more satisfied than the control group with every dimension of 
school life about which they were asked. For example, over half of the 
scholarship parents were very satisfied with the academic quality of the 
school, as compared to one-sixth of the control group. Similarly, 49 percent 
of the scholarship parents expressed the highest satisfaction with “what’s 
taught in school,” as compared to 19 percent of the control group. 

Scholarship students in third, fourth, and fifth grade are less likely to give it a 
failing grade. 

• Parents reported that scholarship students attended smaller schools and were 
being educated in smaller classes. On average, the schools attended by 
scholarship students had 110 fewer students than the schools attended by 
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students in the control group. Furthermore, the classes in which scholarship 
students were being taught had two fewer students on average than the classes 
attended by the control group. Consistent with these findings, 36 percent of 
the scholarship parents reported being “very satisfied” with the class size at 
their school, but only 13 percent of the control group of parents reported a 
similar level of satisfaction. 

• Scholarship students were less likely than the control group to have access to 
a library, cafeteria, nurse’s office, child counselors and special programs for 
non-English speakers and students with learning difficulties. 

• As compared to those in the control group, parents of scholarship students 
were more likely to report that the following were not a serious problem at 
their school: students destroying property, being late for school, missing 
classes, fighting, cheating, and engaging in racial conflict. Twenty two 
percent of the parents in the control group were very satisfied with school 
safety, but almost half of scholarship parents were. 

• Scholarship students were asked to do more homework, parents say. Fifty- 
one percent of the scholarship parents reported that their child had at least an 
hour of homework a day, as compared to 36 percent of the control-group 
parents. Sixteen percent of the control group of parents but only 10 percent of 
the scholarship parents reported that homework was too easy. Scholarship 
students were more likely than control-group students to report having 
difficulty with homework but were less likely to say their work was marked 
and returned to them. 

• Parents of scholarship students report more frequent school communications 
from their child’s school and from their child’s teachers. 

• Scholarship students are considerably more likely than the children in the 
control group to be subject to a dress code and to be required to wear a school 
uniform; they are less likely to be required to obtain a “hall pass” when 
leaving the classroom. 

• Student mobility rates among schools within the school year are the same for 
the scholarship students and the members of the control group. 

• Expulsion and suspension rates were low for both scholarship students and 
members of the control group. No difference between the two groups was 
detected. 

• Using a scholarship reduced somewhat the racial isolation of minority 
students. Eighteen percent of scholarship parents replied that less than half of 
those in their child’s classroom were of minority background, while just 1 1 
percent of the parents in the control group gave this response. Conversely, 37 
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percent of the control-group parents said all the students in the classroom were 
minority, as compared to just 28 percent of the scholarship parents. 

• Parental self reports of their involvement in school and engagement in the 
education of their children was very high and much the same for both 
scholarship parents and those in the control group. 

Making Use of a Scholarship Offer: 

• Seventy-five percent of those offered scholarships made use of them. 

Families who made use of the scholarships were of somewhat higher income 
than those who did not. The expense of a private school was the most 
frequently given reason for not using a scholarship. Failure to pass an 
admissions test was hardly ever mentioned. 

• Over 75 percent of the parents of scholarship users said that among the 
reasons “very important” for their choice of school were school safety, 
academic quality, what was taught in school, and school discipline. The 
school’s location, the children’s friends, and the sports program were least 
frequently mentioned as very important. Religious instruction was mentioned 
as very important by approximately half the parents. 

• Thirty-four percent of scholarship parents said that, apart from their SCSF 
scholarship, they relied on family income to help pay for their private-school 
tuition. Twenty-two percent of the parents said their child had received a 
school scholarship, and 6 percent reported that they the school paid for some 
or, in a few cases, all of their remaining tuition. Sixteen percent of the 
respondents indicated that relatives and friends helped out. 

Participation in the Program: 

Families of scholarship applicants were similar to those eligible for participation 
in terms of income. However, they were more dependant on governmental assistance and 
more likely to be African American. On the other hand, mothers of scholarship students 
had more education and were more likely to report working full-time. 

This is the first opportunity to estimate the impacts of a school choice pilot 
program by means of an evaluation that has all the following characteristics: 

• a lottery that allocated scholarships randomly to eligible applicants; 

• a lottery that was administered by an independent evaluation team that can 
guarantee its integrity; 
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• baseline data on student test performance and family background 
characteristics that were collected from a high percentage of the students and 
their families prior to the lottery; 

• data on a broad range of characteristics that were collected from a high 
percentage of the test group and control group one year later. 
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An Evaluation of the New York School Choice Scholarships Program: 

The First Year 



Over the past few years, legislative proposals have been introduced in Congress 
and many state legislatures. These proposals would offer families vouchers or 
scholarships so that they may choose among a wide range of schools, public and private, 
religious and secular. In 1990 a pilot program giving public students access to secular 
private schools in the City of Milwaukee was enacted by the Wisconsin legislature; in 
1996 the legislature expanded this program to include religious schools. After surviving 
a constitutional challenge in state courts, the program went into effect in the fall of 1998. 
A similar program enacted for Cleveland by the Ohio legislature is beginning its third 
year of operation in the fall of 1998. At the federal level, a pilot program for the District 
of Columbia received congressional approval in 1998 but was vetoed by President Bill 
Clinton. 

Many interest groups, political leaders and policy analysts have debated the 
desirability of continuing and expanding these school choice programs. Supporters of 
school choice assert that: 

• low-income, inner-city children learn more in private schools; 

• a more orderly educational climate in private schools enhances learning 
opportunities; 

• private schools use their limited resources more efficiently; 

• families develop closer communications with schools they themselves choose; 

• school choice reduces the amount of mobility from school to school both 
within the school year and from one year to the next; 

• choice fosters racial and ethnic integration. 1 



These claims are challenged by school choice critics who argue that: 

• any perceived learning gains in private schools are due to the more selected 
nature of private-school families; 

• private schools select out the “best and the brightest,” leaving behind the 
disadvantaged; 

• private-school rules, such as uniforms and dress codes, interfere with a child’s 
creativity; 



• low-income families choose schools more on the basis of location, religious 
affiliation, and sports programs than educational quality; 

• public schools have a broader range of programs to serve needy populations; 

• when choices are available, mismatches often occur and private schools expel 
problem students, adding to the instability of the education of children from 
low-income, inner-city families; 

• private schools balkanize the population into racially and ethnically 
homogenous educational environments . 2 

Much of the debate over school choice has acquired a particularly intense tone in 
part because high-quality information about school-choice programs is limited. Although 
many studies comparing public and private schools have been published, they have been 
criticized for comparing dissimilar populations. Even when statistical adjustments are 
made, it remains unclear whether findings actually describe differences between public 
and private schools or simply differences in the kinds of students and families attending 
them . 3 



The best way to make sure that two populations are similar is to assign individuals 
randomly to treatment and control groups. This procedure is standard in medical 
research. It has also been used recently in a number of educational studies, such as the 
Tennessee Star study that found that smaller classes had positive effects among students 
in kindergarten and first grade . 4 Until now, this type of research design has not been used 
carefully to study the question of school choice. 

In this paper we report outcomes from a randomized experiment made possible by 
the New York School Choice Scholarships Foundation (SCSF), a privately- funded school 
choice program . 5 The program provides the first opportunity to estimate the impacts of a 
school choice pilot program that has the following characteristics: 

• a lottery that allocated scholarships randomly to applicants; 

• a lottery that was administered by an independent evaluation team that can 
guarantee its integrity; 

• baseline data on student test performance and family background 
characteristics that were collected from a high percentage of the students and 
their families prior to the lottery; 

• data on a broad range of characteristics that were collected from a high 
percentage of the test group and control group one year later. 
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In this report, we provide information on the SCSF program one year after 
students started using their school-choice scholarships. Similar reports are planned for at 
least the next two years of the program. 

Our major findings, in summary form, are as follows: 

Outcomes: 

• After one year, students who received a scholarship scored higher in math and 
reading tests. Overall, the differences between all those in grades two through 
five who used their scholarships to attend a private school and the control 
group of students were small — around two percentile points in both subjects. 
However, sizeable differences were observed among those students who were 
in the fourth and fifth grades — four percentile points in reading and six points 
in math. 

• Parents of scholarship users are much more satisfied with their children’s 
education. Nearly half the scholarship users give their school an “A”, while 
only one-eighth of the control group does. Scholarship families were 
substantially more satisfied than the control group with every dimension of 
school life about which they were asked. For example, over half of the 
scholarship parents were very satisfied with the academic quality of the 
school, as compared to one-sixth of the control group. Similarly, 49 percent 
of the scholarship parents expressed the highest satisfaction with “what’s 
taught in school,” as compared to 19 percent of the control group. Scholarship 
students in third, fourth, and fifth grade are less likely to give it a failing 
grade. 

• Parents reported that scholarship students attended smaller schools and were 
being educated in smaller classes. On average, the schools attended by 
scholarship students had 110 fewer students than the schools attended by 
students in the control group. Furthermore, the classes in which scholarship 
students were being taught had two fewer students on average than the classes 
attended by the control group. Consistent with these findings, 36 percent of 
the scholarship parents reported being “very satisfied” with the class size at 
their school, but only 13 percent of the control group of parents reported a 
similar level of satisfaction. 

• Scholarship students were less likely than the control group to have access to 
a library, cafeteria, nurse’s office, child counselors and special programs for 
non-English speakers and students with learning difficulties. 

• As compared to those in the control group, parents of scholarship students 
were more likely to report that the following were not a serious problem at 
their school: students destroying property, being late for school, missing 
classes, fighting, cheating, and engaging in racial conflict. Twenty two 



percent of the parents in the control group were very satisfied with school 
safety, but almost half of scholarship parents were. 

• Scholarship students were asked to do more homework, parents say. Fifty- 
one percent of the scholarship parents reported that their child had at least an 
hour of homework a day, as compared to 36 percent of the control-group 
parents. Sixteen percent of the control group of parents but only 10 percent of 
the scholarship parents reported that homework was too easy. Scholarship 
students were more likely than control-group students to report having 
difficulty with homework but were less likely to say their work was marked 
and returned to them. 

• Parents of scholarship students report more frequent school communications 
from their child’s school and from their child’s teachers. 

• Scholarship students are considerably more likely than the children in the 
control group to be subject to a dress code and to be required to wear a school 
uniform; they are less likely to be required to obtain a “hall pass” when 
leaving the classroom. 

• Student mobility rates among schools within the school year are the same for 
the scholarship students and the members of the control group. 

• Expulsion and suspension rates were low for both scholarship students and 
members of the control group. No difference between the two groups was 
detected. 

• Using a scholarship reduced somewhat the racial isolation of minority 
students. Eighteen percent of scholarship parents replied that less than half of 
those in their child’s classroom were of minority background, while just 1 1 
percent of the parents in the control group gave this response. Conversely, 37 
percent of the control-group parents said all the students in the classroom were 
minority, as compared to just 28 percent of the scholarship parents. 

• Parental self reports of their involvement in school and engagement in the 
education of their children was very high and much the same for both 
scholarship parents and those in the control group. 

Making Use of a Scholarship Offer: 

• Seventy-five percent of those offered scholarships made use of them. 

Families who made use of the scholarships were of somewhat higher income 
than those who did not. The expense of a private school was the most 
frequently given reason for not using a scholarship. Failure to pass an 
admissions test was hardly ever mentioned. 
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• Over 75 percent of the parents of scholarship users said that among the 
reasons “very important” for their choice of school were school safety, 
academic quality, what was taught in school, and school discipline. The 
school’s location, the children’s friends, and the sports program were least 
frequently mentioned as very important. Religious instruction was mentioned 
as very important by approximately half the parents. 

• Thirty-four percent of scholarship parents said that, apart from their SCSF 
scholarship, they relied on family income to help pay for their remaining 
private-school tuition. Twenty-two percent of the parents said their child had 
received a school scholarship, and 6 percent reported that the school paid for 
some or, in a few cases, all of their tuition. Sixteen percent of the respondents 
indicated that relatives and friends helped out. 

Participation in the SCSF program: 

Families of scholarship applicants were similar to those eligible for participation 
in terms of income. However, they were more dependent on government assistance and - 
more likely to be African American. On the other hand, mothers of scholarship students 
had more education and were more likely to report working full-time. 

In the remainder of this report, we describe the program sponsored by the SCSF, 
the data collection, analysis and reporting procedures that the evaluation team employed, 
and detailed findings from the evaluation for the first year of the program. 



Description of the School Choice Scholarships Foundation Program 

In February 1997 SCSF announced that it would provide 1,300 scholarships worth 
up to $1,400 annually for at least three years to children from low-income families 
currently attending public schools. The scholarship could be used to help pay the cost of 
attending a private school, either religious or secular. SCSF received initial application 
forms from over twenty thousand students between February and late April 1997. 

In order to become eligible for a scholarship, children had to be entering grades 
one through five, live in New York City, attend a public school at the time of application, 
and come from families with incomes such that they qualified for the U. S. government’s 
free school lunch program. To ascertain eligibility, students and an adult member of their 
family were asked to attend verification sessions during which family income and their 
child's public-school attendance was documented. 

Because of the large number of families submitting initial application forms, it 
was not feasible to invite all families to verification sessions. To give all families an 
equal chance of participating, a preliminary lottery was used to determine which families 
would be invited to verification sessions. Only families who attended the sessions and 
documented their eligibility were included in the final lottery. 
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The final lottery was held in mid-May 1 997. Mathematica Policy Research 
(MPR) administered the lottery; SCSF announced the winners. Within the parameters 
established by SCSF, all applicants had an equal chance of winning the lottery. SCSF 
decided in advance to allocate 85 percent of the scholarships to applicants from public 
schools whose average test scores were less than the citywide median. Consequently, 
applicants from these schools, who represented about 70 percent of all applicants, were 
assigned a higher probability of winning a scholarship. In the information reported in the 
tables, results have been adjusted by weighting cases differentially so that they can be 
generalized to all applicants who would have come to the verification sessions had they 
been invited, regardless of whether or not they attended a low-performing school. 

Subsequent to the lottery, SCSF assisted families in finding private school 
placements. In mid-September 1997, SCSF reported that places had been found and 
recorded at some 225 private schools for 1,168 scholarship recipients, approximately 75 
percent of all those offered scholarships. 



Evaluation Procedures 

The evaluation procedures that were utilized conform to those used in randomized 
experiments. Baseline data were collected prior to the beginning of the scholarship 
program; one of the conditions for participating in the program was agreement to provide 
confidential baseline and follow-up information. Responses were obtained one year later 
from '83'percent of all families participating in the evaluation. 6 

Collection of Baseline Data 

During the sessions at which eligibility was verified, students were asked to take 
the Iowa Test in Basic Skills (ITBS) in reading and mathematics. Students in 
kindergarten applying for a scholarship for first grade were exempted from this 
requirement. Parents were asked to fill out questionnaires reporting information on their 
satisfaction with the school their child was currently attending, their involvement in their 
child’s education, and their demographic characteristics. These sessions took place 
during March, April, and early May 1997 on Saturday mornings and on vacation days. 
The sessions were held at private schools, where students could take tests in a classroom 
setting. In most cases, private school teachers served as proctors under the overall 
supervision of the staff of Mathematica Policy Research (MPR). 

While the child was taking a test that took more than an hour, responses to 
questionnaires were completed in a separate room by the adult accompanying the child to 
the testing session. This procedure had the advantage of giving administrators the 
opportunity to stress that responses to the questionnaire would be held in strict 
confidence and used for statistical purposes only. It also provided respondents the time 
to complete the questionnaire at leisure and the opportunity to ask any questions 
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concerning the meaning of particular questions. Questionnaires were available in both 
English and Spanish. 

Anticipating that a variety of caretakers might be accompanying children, 
questions were designed in such a way as to allow any caretaker familiar with the child’s 
school experiences to respond to the questions. Although grandmothers and other 
relatives and guardians also filled out the questionnaire, in over 90 percent of the cases 
one of the parents answered the questions. The remainder of the report, for ease of 
presentation, refers to opinions expressed as those of parents. 

In November 1997 MPR and the Harvard Program on Education Policy and 
Governance (PEPG) issued a report that provided information on baseline test scores and 
demographic characteristics of applicants, scholarship winners, those who used their 
scholarships, and those who did not. Because scholarships were allocated by a lottery, 
there were few differences between scholarship recipients and non-recipients; however, 
baseline test scores of non-recipients were somewhat higher (see appendix). Those 
scholarship recipients who made use of the scholarship were more educated and were 
somewhat less disadvantaged in other respects than those who did not make use of the 
scholarship; however, those making use of the scholarship were more likely to be African 
American. Baseline test scores did not differ significantly between those using the 
scholarship and those who did not. 7 

Collection of First-Year Follow-up Information 

To evaluate the effects of the scholarship on students and their families, MPR 
selected at random students from 1,000 families who had received a scholarship and 960 
families who had attended a verification session but had not received a scholarship. 
Procedures used to select the control group are described elsewhere. 8 In April, May and 
June of 1998, these families were invited to attend sessions during which students took 
the ITBS in mathematics and reading, and adult members of their family completed 
surveys that asked a wide range of questions about the educational experiences of their 
oldest child within the age range eligible for a scholarship. Students in grades three, four 
and five were also asked to complete short questionnaires. (See appendix for 
questionnaires). 

Testing and questionnaire administration procedures were similar to those that 
had been followed one year previously. Both the scholarship students and students in the 
control group were tested in locations other than the school they were currently attending. 
Each student’s performance was given a national percentile ranking that varies between 
one and one hundred. The national average is fifty. 

Eighty-three percent of those selected for participation agreed to attend the testing 
and questionnaire sessions held in April and May 1998. This high response rate was 
achieved in part because SCSF conditioned the renewal of scholarships on participation 
in the evaluation; non- scholarship winners selected to become members of the control 
group were compensated for their expenses and told that they could automatically reapply 
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for a new lottery if they participated in these follow-up sessions. The appendix compares 
the characteristics of respondents and non-respondents. 

Data Analysis and Reporting Procedures 

The analysis of the data from the first year of the SCSF program takes advantage 
of the fact that a lottery was used to award scholarships. As a result, it is possible to 
compare two groups of students that were similar, on average, in almost all respects 
except that the members of the control group were not offered a scholarship. 

Since some families did not participate in the testing and questionnaire 
administration sessions held in the spring of 1998, some departure from an ideal 
randomized experiment occurred. However, as is discussed in the Appendix, the 
response rate was unusually high, especially in light of the fact that data were being 
collected from a low-income, inner city population in the nation’s largest city. We have 
made adjustments in the weight given to individual cases in order to take into account 
differential response rates. 9 

This report provides data that help to answer the following two questions, both of 
which have clear policy implications: 

1 . What was the impact of the offer of a SCSF scholarship to a group of low- 
income scholarship applicants, as measured by test scores and as perceived by 
the applicants themselves? 



2. What was the impact of participation in the SCSF program on those families 
who made use of their scholarship to attend a private school? 

The analytical techniques needed to answer each question differ in important 
ways. The first question can be answered straightforwardly by comparing the responses 
of those who were offered a scholarship with the responses of the control group. Because 
scholarships were awarded at random, the two groups may be assumed to be, on average, 
statistically equivalent, save the offer of a scholarship. Any differences between the two 
groups can be attributed to the offer. 10 

To compute program impacts on children’s test scores, we estimated a statistical 
model that took into account students’ scholarship or control-group status, baseline 
reading and math test scores, and variables used to define the randomization process. 
Baseline test scores were included to: 1) adjust for baseline differences between the 
treatment and control groups on the achievement tests and 2) to increase the precision of 
the estimated impacts. 1 To compute program impacts on the parent and student survey 
outcomes we used a similar approach; however, we did not include the baseline test 
scores to predict parent and student responses. 
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